1. FIELD OF THE INVENTION
The present invention relates to an image motion vector sensor for sensing an amount of displacement or movement of an image.
2. DESCRIPTION OF THE PRIOR ART
A conventional sensor for sensing a motion vector of an image has been described, for example, in the JP-A-61-269475.
FIG. 9 illustratively shows the image motion vector sensor, which comprises a first latch 1, a representative point memory 2, a second latch 3, a correlation device 4, an address controller 5, an address switch circuit 6, a cumulative adder 7, a correlation retrieval circuit 8, a correlation validity decision circuit 9, and a decision circuit 10.
A description will now be given of an image motion vector sensor employing the conventional correlation computing device configured as above.
First, a motion vector of an image will be described. FIGS. 8A, 8B and 8C respectively show an image displayed at a point of time and an image developed after a field or frame. When an image undergoes a translational displacement as above due to a movement of an imaging device or the like, the amount of the translational displacement of the image as indicated by an arrow in a FIG. 8C is represented by a vector, which is called a motion vector.
Next, a method for sensing the image motion vector will be described.
A so-called representative point matching method is most commonly employed so as to sense a motion vector. According to the representative point matching method, first, as shown in FIG. 10A, an entire image plane is divided into a plurality of sense domains (6 domains are illustrated), and each sense domain is further divided into a plurality of motion vector sense regions (9 regions are illustrated), each of the regions having a representative point in it. FIG. 10B shows a motion vector sense region including a representative point and pixels around the representative point.
Then, the image motion vectors are sensed in respective sense domains by finding, with the correlation calculation between the image data of the previous an the present field (or frame), the location of a pixel in the present field (or frame) to which an image has displaced from the location of the representative point in the previous field (or frame) the image having been located. Finally, a single motion vector representing an entire image plane is decided amongst the motion vectors sensed in respective sense domains.
Referring to FIGs. 10A and 10B, a further detailed description of the representative point matching method will be given.
First, an absolute difference between the image data S.sub.n-1 (X.sub.k, Y.sub.k) of the representative point at a coordinate (X.sub.k, Y.sub.k) in the (n-1)th field (or frame) and the image data S.sub.n (X.sub.k+i, Y.sub.k+j) of a pixel at a coordinate (X.sub.k+i, Y.sub.k+j) around the representative point within the k-th moving vector sense region in the n-th field (or frame) is calculated, where k is the serial number for appointing a representative point, i and j are coordinate differences between the representative point and the pixel of attention, where the variable range of i and j correspond to the range within which a motion vector is sensed, and n denotes discrete time. The above difference calculation is performed between the representative point and each of the surrounding pixels in the respective motion vector sense regions. Then, the absolute differences, calculated for each pixel at the location having the same coordinate values on the relative coordinates fixed to the representative point in the respective motion vector sense regions, are added together. In the case of FIG. 10A, as there are 9 representative points, the above cumulative addition is carried out for 9 absolute difference data on each pixel. This cumulative addition results in a set of correlation values P(i, j), the number of which equals to the number of pixels included in one motion vector sense region.
The arithmetic operations described above can be written in the following formula: EQU P(i,j)=.SIGMA..vertline.S.sub.n-1 (X.sub.k, Y.sub.k)-S.sub.n (X.sub.k+i, Y.sub.k+j) .vertline. (1)
where .SIGMA. indicates a summation operation within the variable range of suffix number k, which varies from 1 to the number of the representative points in one sense domain, 9 in the case of FIG. 10A.
The correlation values P(i, j) are thus obtained, and a small P(i, j) value indicates a high correlation, a high P(i, j) value a low correlation. Since the input signals are of the two pictures the one of which has resulted from a translational displacement of the other, there is only one state in which these two pictures can overlap each other. The overlapping state of the two pictures is sensed by finding the coordinate (i, j) where P(i, j) has a minimum value. Then, the image motion vector is given as the coordinate value (i, j). Accordingly, the motion vector is determined with the following operation: EQU V(i, j)=(i, j) for min{P(i, j)} (2)
where "min" denotes an operator for selecting the minimum.
A method for sensing the motion vector with above mentioned arithmetic operations is called the representative point matching method, by which a method vector for each domain is sensed.
Next, referring to FIGS. 9, 10A and 10B, a description will be given of an image motion vector sensor employing the conventional correlation computing device.
Image data at the respective representative points in a field or frame are fetched into a first latch 1 in response to a timing pulse LP1 so as to be written at an appropriate timing in the representative point memory 2 at addresses associated with the respective representative points. Thereafter, in the subsequent field or frame, the system computes correlation between image data at respective pixels in a motion vector sense regions associated with the respective representative points and the image data at the representative points of the previous field beforehand stored in the representative point memory 2, thereby sending the result to the cumulative adder 7. The cumulative adder 7 adds together the correlation data which are associated with the same coordinate positions with the representative point set as a reference in respective motion vector sense regions. When the cumulative addition of the above data is completed for all motion vector sense regions, the correlation retrieval circuit 8 checks the cumulative values kept in the cumulative adder 7 to decide a position where the correlation develops a highest value. That is, with the position of the representative point set as the reference, the position (address) of the highest correlation value is obtained as a motion vector Furthermore, based on a distribution (a mean value, a minimum value, a maximum value, a gradient, or the like) of the correlation values around the representative point, the correlation validity decision circuit 9 judges to decide whether or not the motion vector attained from the correlation computation is valid.
Above-mentioned sensing of image motion vectors and decision of correlation validity are, as shown in FIG. 10A, carried out for a plurality of sense domains, and result in, for every image plane, a plurality of motion vectors which are decided to be valid.
And, amongst valid motion vectors sensed, a single motion vector that represents an entire image plane is decided by a judge circuit 10. In order to make the above decision, a method which is often employed is that a mean value with respect to the valid motion vectors sensed in one image plane or a median of the valid motion vectors is construed as the motion vector representing an entire image plane.
Since the operation above is carried out for each field or frame, in order for the system to save image data of a representative point in the subsequent field or frame while conducting the correlation computation, where is disposed the first latch 1. Moreover, the second latch 3 is used to keep image data at the representative point while correlation is being established between image data at a representative point in the previous field or frame and image data around the representative point in the present field or frame.
In a portion enclosed with broken lines of FIG. 9, a portion in which a motion vector is sensed through the correlation computation constitute a motion vector sense section 11, whereas a portion in which a motion vector representing an entire image plane is obtained by use of motion vectors attained in respective sense domains and correlation information of the vectors attained by the motion vector sense section 11 forms a first vector decision section 12.
In the constitution above, however, depending on a state of the input image signal, the number of sense domains in which the vector can be sensed varies. In the worst case, the motion vector is rarely sensed, and in other case, the motion vector can be sensed or cannot be sensed such that the output motion vectors are only sporadically generated. This leads to a problem that the sensed motion vectors are discontinuous with respect to time.